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Executive  Summary 

EVALUATION  OF  DEMAND  PREDICTION  TECHNIQUES 


Reparable  spares  of  aircraft  components  constitute  an  important  management 
item  for  the  Air  Force,  amounting  to  a  computed  budget  requirement  of 
$4,077  billion  in  FY85. 

Allocation  of  this  investment  across  items  is  critical  to  the  readiness  and 
sustainability  of  weapon  systems.  Proper  allocation,  in  turn,  depends  on  solving  the 
statistical  problem  of  estimating  item  demand  rates  and  variances. 

Using  historical  Air  Force  data,  we  have  compared  the  performance  of  various 
estimating  procedures,  including  the  one  used  in  the  Air  Force  Recoverable 
Consumption  Item  Requirements  (D041)  system,  which  computes  item  Peacetime 
Operating  Stock  (POS)  requirements.  To  be  consistent  with  Air  Force  orientation 
toward  weapon  system  management,  aircraft  availability,  an  aspect  of  readiness, 
served  as  the  measure  of  performance. 

Our  major  conclusions: 

•  For  estimating  demand  rates,  exponential  smoothing  techniques  are. 
superior  to  today’s  D041  moving-average  rates  and  are,  in  fact,  the  best 
techniques  of  those  we  studied. 

•  Use  of  a  Poisson  or  constant  variance-to-mean  ratio  (VMR)  leads  to  poor 
allocation  of  resources.  Treating  the  VMR  as  a  power  function  of  the  mean 
demand  rate  is  preferable. 

Incorporating  these  techniques  into  the  D041  computation  would  be  a  simple 
matter,  and  the  Air  Force  would  benefit  significantly.  When  compared  to  current 
AFLC  policy,  the  proposed  techniques  showed  a  reduction  in  backorders  of  over 
50  percent  for  the  F-16  and  the  A-10  weapon  systems. 
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CHAPTER  1 


INTRODUCTION 


Reparable  items  constitute  an  important  management  problem  for  the 
Air  Force.  In  FY85,  the  computed  budget  requirement  for  these  items  was 
$4,077  billion,  and  allocation  of  this  investment  across  items  has  a  critical  effect  on 
the  availability  of  weapon  systems. 

Few  studies  of  demand  prediction  pertaining  specifically  to  the  Air  Force  have 
been  conducted  in  recent  years.  The  Stevens  and  Hill  [1]  study  is  the  basis  for  much 
of  Air  Force’s  present  policy.  A  recent  study,  Sherbrooke  [2],  estimated  that  ex¬ 
ponential  smoothing  with  a  constant  of  .4  applied  to  quarterly  data  could  produce  a 
39-percent  reduction  in  squared  error  and  a  12-percent  reduction  in  average  absolute 
error  over  the  8-quarter  moving  average  now  in  use.  A  modification  of  the  equation 
for  predicting  the  variance-to-mean  ratio  (VMR)  was  also  proposed. 

The  weapon  system  approach  of  the  present  study  distinguishes  it  from  earlier 
ones.  The  data  in  Sherbrooke  [2]  came  from  a  stratified  random  sample  of 
1,027  items  from  the  Air  Force  inventory.  With  the  data  on  all  reparable  items  for 
each  of  several  weapon  systems,  it  is  possible  for  availability  to  serve  as  the  measure 
of  performance.  This  is  more  meaningful  than  such  measures  as  squared  error  or 
average  absolute  error,  because  availability  is  the  Air  Force’s  objective  when  it 
allocates  funds  to  spare  parts. 

Another  reason  to  perform  this  study  was  to  see  whether  the  recommendations 
for  predicting  demand  were  consistent  across  several  weapon  systems.  If  the 
recommendations  varied  between  one  weapon  system  and  another,  their  utility  and 
validity  would  be  questionable.  Moreover,  though  quarterly  Recoverable 
Consumption  Item  Requirements  (D041)  data  serve  as  the  primary  source,  as  in  the 
earlier  study,  it  was  important  to  obtain  some  transaction  data  from  an  individual 
base.  This  information  yields  the  interval  between  demands  —  not  quarterly 
summaries  —  and  thus  tells  us  more  about  the  "physics”  of  the  demand  process. 

Section  2  describes  the  demand  prediction  experiment,  beginning  with  defini 
tions  of  several  candidate  techniques  for  predicting  mean  demand  and  the  VMR.  We 
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discuss  the  availability  criterion,  the  data  provided  by  the  D041  system,  and  the 
prediction  process  for  each  technique  {a  "technique”  being  defined  as  a  process  for 
predicting  the  mean  demand  and  the  VMR).  Finally,  we  describe  the  evaluation 
procedure  and  compare  the  performance  of  the  various  techniques. 

Chapter  3  presents  the  results,  Chapter  4  the  major  conclusions.  Detailed 
results  are  reported  in  the  appendices. 
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CHAPTER  2 


DESCRIPTION  OFTHE  EXPERIMENT 


The  objectives  of  this  study  are  to  find  demand  prediction  techniques 
that:  (1)  achieve  a  high  level  of  system  availability  in  the  future  and  (2)  predict  a 
level  of  system  availability  similar  to  the  level  achieved. 

The  candidate  procedures  for  predicting  mean  demand  fall  into  the  four 
categories  shown  in  Table  2-1.  Let  M  be  the  predicted  mean  demand,  D(l)  the 
demand  in  the  most  recent  quarter,  D(2)  the  demand  in  the  preceding  quarter,  etc. 
For  exponential  smoothing,  an  example  is  shown  where  the  smoothing  constant,  A, 
is  set  at  .4.  (In  Sherbrooke  [2]  a  value  of  .4  on  quarterly  data  appeared  best.)  The  Air 
Force  now  uses  the  second  procedure,  a  moving  average,  with  N  =  8.  This  weights 
each  of  the  last  8  quarters  equally  and  gives  no  weight  to  earlier  quarters. 

The  Bayesian  procedure  is  the  "objective  Bayes”  technique  that  was  applied  at 
George  Air  Force  Base  in  1965  and  documented  in  Feeney  and  Sherbrooke  [3].  The 
basic  idea  is  that  the  two  parameters  of  a  single  gamma  prior  probability 
distribution  are  estimated  from  past  demand  on  all  items.  Bayes  theorem  is  then 
used  to  combine  this  pricr  distribution  with  specific  data  about  an  item  to  obtain 
posterior  distributions  that  will  differ  by  item.  The  important  difference  between 
the  Bayes  procedure  and  the  two  previous  procedures  is  that  the  Bayesian  approach 
yields  a  probability  distribution  —  instead  of  a  point  estimate  —  of  demand.  There 
is,  therefore,  no  need  to  develop  a  variance-estimating  procedure  for  the  Bayes 
technique. 

Finally,  it  is  possible  to  use  higher-order  procedures  where  both  a  trend  and  a 
mean  value  are  estimated.  We  followed  a  procedure  called  "Holt  linear  estimation,” 
which  is  described  in  Makridakis  and  Hibon  [4];  it  consists  basically  of  second-order 
exponential  smoothing.  The  first  parameter  is  the  same  as  the  first-order 
exponential  smoothing  parameter  and  is  set  at  .4,  as  above.  The  second  parameter  is 
a  trend  value  that  we  set  at  .5  after  some  trial  and  error.  However,  we  restricted  the 
rate  of  growth  for  trend  to  eliminate  absurd  inferences  or  estimates  of  negative 
demand.  For  example,  the  absolute  value  of  the  trend/quarter  was  limited  to 
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15  percent  of  the  mean  value,  so  that  over  4  periods  the  mean  could  increase  or 
decrease  by  60  percent  at  most. 

TABLE  2-1 

PROCEDURES  FOR  PREDICTING  MEAN  DEMAND 


1.  Exponential  smoothing 

M  =  AD(1)  +  A(1  -  A)D(2)  +  A(1  -  AR  D(3) . . . 
Example:  A  =  .4 

M  =  4D(1)  +  .240(2)  +  .  144D(3)  . . 

2.  Moving  average 

M  =  (D(1)  +  D(2) .  .  .  +  D(N))/M 
Example:  N  =  8 

Prediction  =  125D(1)  +  125D(2) .  +  125D(8) 

3.  Bayes 

Prediction  is  a  combination  of  observed  demand  and 
the  prior  probability  distribution 

Estimate  of  variance  provided 

4.  Higher-order  procedures  with  trend  also  estimated 


The  earlier  work  for  Air  Force  Logistics  Command  (AFLC)  in  Sherbrooke  [2] 
made  it  clear  that  it  is  important  to  estimate  the  VMR  as  well  as  the  mean.  Four 
types  of  candidate  procedures  are  considered  in  Table  2-2.  The  first  is  a  power 
relation,  where  the  VMR  is  an  increasing  function  of  the  mean.  Here  the  mean  is 
expressed  as  an  annual  value.  If  the  mean  is  expressed  over  any  other  time  period, 
the  exponent  is  unchanged,  but  the  multiplicative  constant  is  altered.  The 
expression  shown  here  is  very  similar  to  the  expression  derived  in  the  earlier  AFLC 
study  and  does  well  in  this  experiment,  too.  Note  that  the  independent  variable,  the 
mean,  is  not  measured  over  the  repair  time.  That  would  understate  observed 
variability,  because  variability  is  caused  not  by  the  duration  of  repair  time,  but  the 
drift  in  the  mean  value  over  time.  We  shall  return  to  this  subject. 
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TABLE  2*2 


PROCEDURES  FOR  PREDICTING  VARIANCE 

1.  Power  relations 

Example:  VMR  =  1  +  ,14M'5 

2.  Based  on  sample  variance  from  past  data 

3.  Bayesian 

4.  Combinations  of  Techniques  1, 2,  and  3 


Another  procedure  is  to  estimate  the  item  variance  from  the  variance  computed 
from  the  earlier  data  (or  from  the  mean  absolute  deviation,  as  in  exponential 
smoothing).  A  third  type  of  procedure  is  the  Bayesian,  which  automatically  provides 
an  estimate  of  variance.  Finally,  recent  research,  as  in  Winkler  and  Makridakis  [5], 
suggests  that  combinations  of  procedures  may  outperform  individual  procedures.  In 
our  application,  a  general  power  relation,  such  as  Procedure  1,  in  combination  with  a 
Procedure  2  that  is  sensitive  to  extreme  variation  in  the  history  of  an  item,  may 
outperform  either  procedure  alone. 

A  major  difference  between  this  study  and  earlier  studies  of  demand  is  the 
evaluation  mechanism.  Most  prediction  studies  use  some  model-independent 
measure  to  evaluate  accuracy.  The  problem  is  that  there  are  many  candidates,  and 
it  is  hard  to  decide  which,  if  any,  is  appropriate.  Average  absolute  error  is  a  common 
measure,  but  should  the  error  on  a  very-high-demand  item  be  divided  by  some  factor 
before  adding  it  to  the  error  on  a  low-demand  item?  Should  the  same  error  on  two 
items  that  are  widely  different  in  unit  costs  be  weighted  equally?  If  the  error  is  twice 
as  large,  should  the  penalty  be  twice  as  heavy  —  or  should  it  be  heavier,  as  in  a 
squared-error  criterion?  Is  it  reasonable  to  assign  as  much  weight  to  an  over 
prediction  error  as  to  an  underprediction  error?  Many  economic  studies  use 
percentage  error,  but  many  of  our  items  have  a  "true  demand”  of  zero.  This  causes 
problems  as  a  divisor. 
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Our  view  is  that  end-item  availability  should  be  used,  because  the  objective  of 
demand  prediction  on  a  group  of  items  is  to  maximize  availability  for  a  specified 
budget.  Availability  is  easily  calculated,  as  shown  in  the  formula  in  Table  2-3.  We 
need  to  compute  the  expected  backorders  (EBOs)  for  a  given  item  and  divide  by  the 
number  of  end  items,  N  (e.g.,  aircraft).  Subtracting  from  1  gives  the  expected 
fraction  of  the  fleet  that  is  not  down  for  lack  of  that  item  (under  the  assumptions  that 
backorders  are  randomly  distributed  across  aircraft  and  the  number  of  backorders 
for  any  items  is  small  compared  with  the  size  of  the  fleet).  Multiplying  by  a  similar 
expression  for  every  item  gives  the  fraction  of  the  fleet  that  is  available. 

TABLE  2-3 

EVALUATION  OF  TECHNIQUES 


Model-independent  measures 

Average  absolute  error 
Mean-squared  error 
Percentage  error 

Model-dependent  measure 

Availability  =  (1  -  EBO(1)/N)[1  -  EBO(2)/N] . . . 


The  data  for  this  demand  prediction  experiment  were  taken  from  16  quarters  of 
D041  data.  We  drew  on  the  first  12  quarters  for  historic?!  data  and  used  the  last 
4  quarters  to  evaluate  accuracy  of  prediction.  C-5s  were  selected  because  they 
operate  from  only  two  home  bases.  Worldwide  demand,  as  drawn  from  the  D041,  is 
therefore  not  too  different  from  base-level  demand,  although  demands  do  occur  at 
other  en  route  bases  during  flight.  The  data  used  in  the  calculations  are  listed  in 
Table 2-4:  base-level  demand  by  quarter,  unit  cost,  not-reparable-this-station 
(NRTS)  demand  (which  goes  to  depot),  and  average  base  and  depot  repair  times  by 
item.  All  nonengine,  first-indenture  items  (line  replaceable  units)  with  some 
demand  —  there  were  560  such  items  —  were  used. 

Data  from  the  C-5  engine,  A-10  airframe,  and  the  F-16  engine/airframe  were 
aiso  used  in  further  tests.  These  tests  are  described  in  Appendices  A  through  E.  The 
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average  repair  time  by  item  is  the  sum  of:  (l)the  base  repair  time  multiplied  by 

(1  —  NRTS)  and  (2)  the  order- and -ship  time  plus  the  average  depot  delay  caused  by 

• 

not  having  an  item  on  the  shelf,  multiplied  by  the  NRTS  fraction.  We  assumed 
an  order-and-ship  time  of  2  weeks.  The  average  depot  delay  is  unknown,  but  it  will 
tend  ho  be  longer  if  the  depot  repair  time  is  longer.  As  an  arbitrary  —  but 
reasonable  -  procedure,  we  assumed  that  the  depot  delay  would  be  .3  times  the 
depot  repair  time.  (We  also  used  .1  with  qualitatively  similar  results.) 


TABLE  2-4 
DATA 


Source: 

D041 

#  Items: 

560 

System: 

C-5  nonengine,  first  indenture 

Elements: 

1 .  Worldwide  base  demand  by  item  by  quarter 

2.  Unit  cost,  NRTS  value,  average  base  and  depot  repair  times 
by  item 

H, story: 

12  quarters 

Predict: 

4  quarters 

The  prediction  process,  summarized  in  Table  2-5,  consists  of  taking  the 
12  historical  quarters  of  demand  data  and  estimating  the  mean  and  VMR  of  demand 
for  each  item.  The  mean  demand  over  the  repair  time  and  the  VMR  are  used  to 
estimate  the  two  parameters  of  a  negative  binomial  distribution  of  demand.  This 
demand  distribution  is  used  to  calculate  the  EBOs  as  a  function  of  the  stock  level  for 
each  item.  Then  these  backorder  functions  are  used  with  unit  cost  to  marginally 
allocate  a  fixed  investment  optimally  across  the  560  items.  The  result  is  a  set  of 
stock  levels  for  each  technique  and  an  overall  predicted  availability.  In  the  case  of 
the  Holt  second-order  smoothing  that  predicts  trend,  stock  levels  from  the  marginal 
allocation  will  be  different  in  each  predicted  quarter. 
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TABLE  2-5 


1.  Demand  data  by  item  for  12  quarters 

2.  NRTS  and  average  repair  time  by  item 

3.  Budget  to  allocate  across  items 

4.  Specification  of  demand  prediction  techniques 


Process 


1.  Calculate  EBOs  as  a  function  of  stock  level  for  each  prediction 
technique  and  item 

2.  Use  marginal  analysis  to  allocate  investment  across  items  to 
maximize  the  predicted  availability  for  the  budget  specified 


Output  by  prediction  technique 

1 .  Stock  level  for  each  item 

2.  Predicted  availability 


We  next  evaluated  the  prediction  techniques  as  summarized  in  Table  2  6.  The 
inputs  for  the  evaluation  are  the  actual  demands  during  the  last  4  quarters,  not  used 
for  prediction,  and  the  stock  levels  by  item.  Unfortunately,  we  do  not  know  the  day 
on  which  each  demand  occurred,  only  the  total  quarterly  demand.  We  therefore  use 
a  random  number  generator  to  draw  the  day  during  the  quarter  for  each  demand. 
On  the  basis  of  the  item  data  on  NRTS  rate  and  repair  times,  we  compute  the  day  on 
which  repair  of  the  item  will  be  completed.  (The  average  repair  time  for  the  item  is 
used,  not  a  draw  from  a  probability  distribution  with  that  mean.)  The  same 
sequence  of  demands  and  repair  times  is  used  for  all  techniques. 

The  output  of  this  activity  is  an  achieved  availability  for  each  technique.  The 
availability  is  computed  daily  and  averaged  for  each  of  the  4  quarters  and  for  the 
entire  year.  Availability  with  cannibalization  —  consolidation  of  shortages  onto  the 
smallest  number  of  aircraft  —  is  also  computed. 
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TABLE  2-6 


EVALUATION  PROCESS 


Input 

1.  Total  demand  by  item  for  each  of  4  prediction 
quarters 

2.  Item  stock  levels  by  technique  (and  quarter  when 
the  technique  estimates  the  trend) 

Process 

1.  Draw  from  a  uniform  distribution  the  day  (during 
the  quarter)  on  which  each  demand  occurs 

2.  Using  the  item's  average  repair  time,  determine 
the  day  that  each  repair  is  completed 

Output 

Average  achieved  availability  by  technique 


•  CHAPTER 3 


RESULTS  OF  THE  EXPERIMENT 


Detailed  results  are  reported  in  the  appendices.  Here  we  display  two 
important,  general  results  that  have  been  confirmed  in  all  of  our  work  on  demand 
prediction.  We  term  them  "Fundamental  Truths  of  Demand  Prediction.” 

Finding  # 1 :  Demand  in  any  period  is  more  highly  correlated  with  recent 
demand  than  with  earlier  demand. 

The  supporting  evidence  is  to  be  found  in  Table  3-1.  The  first  two  columns  are 
taken  from  Sherbrooke  [2].  In  that  study,  we  computed  the  correlation  between 
demand  in  quarter  A  with  demand  in  quarter  B  for  a  group  of  items  drawn  from  a 
stratified  random  sample.  For  16  quarters  of  data,  the  quarters  are  separated  by 
1  quarter  in  15  correlations,  by  2  quarters  in  14  correlations,  etc.  The  first  column  of 
correlations  shows  that  with  1,027  recoverable  items  selected  at  random, 
correlations  decline  smoothly  from  .966  to  .845,  as  the  number  of  quarters  separating 
the  quarters  analyzed  increases  from  1  to  15. 

The  specific  numbers  are  not  important;  they  depend  on  the  scatter  in  demand 
rates  across  items.  The  demand  in  each  of  2  quarters  across  a  group  of  items  will 
appear  to  be  more  highly  correlated  when  the  item  demand  rates  are  scattered  more 
widely.  Thus,  if  we  exclude  the  28  items  for  which  there  were  more  than 
1,000  demands  in  the  first  8  quarters  and  compute  the  correlations  for  the  remaining 
999  items,  the  correlations  fall  from  .924  to  .454,  as  the  number  of  quarters 
separating  the  quarters  analyzed  increases  from  1  to  15. 

The  result  is  similar  with  items  belonging  to  a  single  weapon  system.  The 
correlations  in  demand  for  recoverable  items  in  the  A- 10  airframe  and  the 
F  16  engine/airframe  are  shown  in  the  last  two  columns  of  Table  3-1.  Though  the 
correlations  of  the  A- 10  do  not  decrease  uniformly  as  the  number  of  quarters  apart 
increases,  the  general  trend  is  downward.  Though  flying  hours  for  both  the  A- 10  and 
F-16  increase  over  the  16  quarters,  the  correlation  coefficient  is  not  affected;  that  is. 
the  slope  of  the  relationship  is  changed  but  not  the  distances  from  the  regression 
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line.  Correlations  based  on  demand  per  flying  hour  (see  Appendix  B)  confirm  these 
results. 


TABLE  3-1 

FUNDAMENTAL  TRUTH  #1  OF  DEMAND  PREDICTION 


Number  of 
quarters 
apart 

1,027  items 

999  items 

480  items 
(A- 10) 

720  items 
(F-16) 

1 

.966 

.924 

972 

966 

2 

.956 

.904 

.953 

944 

3 

949 

.886 

.948 

.939 

4 

.938 

.869 

941 

935 

5 

.930 

.833 

935 

919 

6 

914 

.807 

931 

899 

7 

910 

.791 

923 

888 

8 

.901 

.752 

921 

887 

9 

902 

.718 

912 

.862 

10 

887 

.662 

893 

.836 

11 

881 

634 

917 

834 

12 

880 

594 

.944 

824 

13 

856 

523 

937 

.811 

14 

851 

.472 

932 

780 

15 

.845 

454 

921 

.740 

We  confirmed  these  results  using  data  other  than  quarterly  data  summed  over 
all  bases  for  a  weapon  system,  as  provided  by  D041.  The  day  on  which  each  demand 
for  each  item  occurred  was  obtained  for  a  single  air  base  —  England  Air  Force  Base, 
Mississippi  —  and  a  single  weapon  system,  the  A- 10.  These  data,  aggregated  into 
demand  for  2-week  periods,  are  shown  in  Table  B-4  of  Appendix  B.  It  turns  out  that 
the  correlational  pattern  is  similar,  even  for  shorter  periods  and  an  individual  air 
base. 
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One  problem  with  the  correlations  in  Table  3-1  is  the  impossibility  of 
determining  whether  the  behavior  observed  applies  to  all  items  or  to  only  a  few. 
Correlations  computed  over  time  for  individual  items  in  Appendix  B  show  that,  for  at 
least  25  percent  of  the  items,  correlations  are  positive  and  significant  when  data  are 
drawn  from  adjoining  periods. 

The  implication  is  that  demand  prediction  techniques  should  allocate  more 
weight  to  more  recent  data,  because  mean  demand  rates  change  over  time.  As  we 
have  seen,  that  is  why  the  most  successful  power  curve  expressions  for  the  VMR  are 
based  on  the  annual  mean,  not  the  mean  over  the  repair  time.  Exponential 
smoothing  is  an  example  of  a  demand  prediction  technique  in  which  more  recent 
data  are  weighted  more  heavily. 

Finding  #2:  Both  variance  and  mean  of  demand  must  be  estimated. 

Table  3-2  shows  the  results  of  assuming  Poisson  demand  where  the  variance 
equals  the  mean  (VMR  =  1).  With  an  $80  million  budget,  predicted  availability  is 
.986,  but  achieved  availability  is  only  .566.  Furthermore,  that  .566  availability  is 
far  below  the  availability  of  .828  that  we  could  achieve  by  predicting  the  variance  to 
mean  from  a  power  curve  relation  based  on  the  mean.  With  that  power  curve 
relation,  an  availability  of  about  .566  could  be  achieved  with  an  investment  of  only 
$63  million,  a  saving  of  about  20  percent  in  investment.  Results  are  similar  when 
the  budget  amounts  to  $100  million.  More  detail  is  provided  in  Appendix  A. 

TABLE  3-2 

FUNDAMENTAL  TRUTH  #2  OF  DEMAND  PREDICTION 


Example 

Predicted 

availability 

Achieved 

availability 

$80M  budget 

VMR  =  1 

.986 

566 

VMR  =  1  +  14M5 

771 

828 

$100M  budget 

VMR  =  1 

999 

622 

VMR  =  1  +  14M5 

911 

913 
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We  have  reached  seven  findings  that  —  though  less  fundamental  —  have 
proved  consistently  true  in  all  of  our  works  so  far: 

1.  Exponential  smoothing  is  the  best  technique  for  pred’ . ting  mean  demand. 
The  best  value  for  the  smoothing  constant,  when  appLed  to  quarterly  data, 
appears  to  be  about  .4  (values  from  .3  to  .5  give  similar  results).  In  the 
earlier  AFLC  study,  Sherbrooke  [2],  exponential  smoothing  with  a 
constant  of  .4  produced  a  12-percent  reduction  in  absolute  error  and  a  30- 
percent  reduction  in  squared  error  when  compared  with  an  8-quarter 
moving  average,  the  current  procedure.  In  the  present  study,  exponential 
smoothing  and  the  8-quarter  moving  average  achieve  about  the  same 
availability  with  a  $100  million  budget;  with  an  $80  million  budget, 
exponential  smoothing  achieves  .83  availability,  as  compared  with  an 
achieved  availability  of  .69  with  an  8-quarter  moving  average. 
Appendix  A  includes  more  extensive  details. 

2.  When  demand  per  program  element  (e.g.,  flying  hours)  is  more  stable  over  a 
base  period  than  demand  itself,  it  should  be  used  in  predictions.  Often,  the 
program  element  is  not  in  the  D041  record  or  is  zero  for  some  quarters  with 
positive  demand,  or  has  apparent  errors  that  lead  to  a  less  stable  data 
series.  For  example,  of  933  items  on  the  F-16,  there  are  774  for  which 
program  element  data  can  be  used,  but  only  511  for  which  demand  per 
program  element  is  more  stable.  For  the  other  422  items,  demand  per 
quarter  is  more  stable  and  should  be  used.  This  question  is  addressed  in 
detail  in  Appendices  B  and  D. 

3.  Data  should  be  screened  for  outliers.  The  best  procedure  is  to  identify 
probable  data  errors  and  bring  them  to  the  attention  of  the  item  manager. 
In  some  cases,  it  may  be  necessary  not  only  to  detect  probable  errors  but 
also  to  make  a  best  guess  as  to  the  correct  value.  A  good  correction 
procedure  that  improves  predictions  for  a  few  items  in  every  weapon 
system  and  never  degrades  the  predictions  is  the  following:  Find  the 
quarter  with  the  greatest  demand  and,  if  that  value  is  more  than  5  times  as 
large  as  the  next  highest  demand  and  greater  than  10  demands,  replace 
that  value  by  the  average  over  the  remaining  11  quarters.  The  only 
exception:  If  the  largest  value  is  in  the  last  quarter,  do  not  use  the  screen. 
Of  course,  the  choices  of  5  times  and  10  demands  are  arbitrary,  but  a  single 
quarter  with  a  very  large  value  is  suspicious,  as  either  a  data  error  or  a 
one-time  special  requirement. 

This  screening  procedure  prevents  the  prediction  from  being  overiy 
sensitive  to  one  very  large  (and  probably  erroneous)  data  point. 
Conversely,  we  want  to  prevent  the  prediction  from  being  confused  by  an 
absurdly  low  data  value.  For  example,  exponential  smoothing  is  most 
sensitive  to  the  last  data  point.  With  the  recommended  smoothing 
constant  of  .4,  40  percent  of  the  demand  prediction  weight  comes  from  the 
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last  data  point.  Suppose  that  every  quarter  of  demand  history  has  a 
positive  program  element  and  more  than  100  demands.  The  one  exception 
is  that  the  data  record  for  the  last  quarter  shows  a  demand  of  zero  (but  a 
positive  program  element).  Since  underprediction  of  a  very  high-demand 
item  can  lead  to  severe  degradation  of  the  system,  it  makes  sense  to  replace 
the  zero  with  something  like  the  previous  data  point  or  the  average  of 
several  previous  data  points. 

In  fact,  there  are  two  such  items  in  the  F-16  data  set  (stock  numbers 
2840011253855  and  2840011253856).  If  such  a  screening  procedure  is  not 
used,  availability  is  nearly  zero  for  all  prediction  techniques  with  a  2-year 
planning  horizon.  Note  that  this  procedure  is  recommended  only  when  the 
last  demand  value  is  zero,  previous  demand  values  have  been  large,  and 
the  program  element  is  positive.  By  contrast,  zero-demand  values  in 
earlier  quarters  are  more  likely  to  be  accurate  (there  has  been  more  time  to 
correct  the  data  record),  and  they  have  less  effect  on  the  prediction. 

4.  Techniques  for  predicting  the  trend  were  unsuccessful.  This  is  a  little 
surprising.  We  have  ample  evidence  that  demand  rates  change  over  time, 
but  we  cannot  predict  the  trend  of  this  change.  As  shown  in  Table  A-2  of 
Appendix  A,  the  Holt  second-order  exponential  smoothing  techniques  are 
among  the  worst  (availability  of  .693  versus  availability  of  .828  from  the 
recommended  technique),  and  the  error  increases  as  we  move  further  into 
the  future.  Of  course,  this  does  not  prove  the  impossibility  of  predicting 
trend.  All  we  can  say  is  that  we  have  been  unable  to  find  a  procedure  that 
improves  the  predictions  by  using  the  trend.  A  simpler  problem  is  to 
predict  not  the  size  of  the  trend  but  the  direction.  Some  unsuccessful 
attempts  to  do  this  are  reported  in  Appendix  C. 

At  this  point,  there  is  no  evidence  that  combination  techniques  improve 
predictive  capability.  In  the  only  combination  we  used  (see  Appendix  A), 
results  for  the  $100  million  target  in  Table  A-3  were  improved  slightly,  but 
the  results  for  the  $80  million  target  in  Table  A-2  were  not.  The  matter 
should  be  explored  further. 

5.  Bayesian  techniques  are  fairly  good,  but  not  the  best  available.  The 
objective  Bayes  procedure  assumes  that  before  any  data  on  an  individual 
item  are  processed,  the  probability  distribution  for  demand  is  the  same  for 
every  item.  We  did  make  one  alteration  in  the  objective  Bayes 
procedure:  For  any  item  with  extremely  heavy  demand  (more  than 
1,200  demands  in  the  First  12  quarters),  the  Bayesian  procedure  was  not 
used,  because  it  has  a  tendency  to  pull  down  the  posterior  demand  mean  to 
an  unreasonably  low  value. 

The  Bayesian  techniques  yielded  the  highest  achieved  availability  for  a 
$100  million  target  for  the  C  5  airframe  (see  Table  A  3  in  Appendix  A)  and 
high  availabilities  for  an  $80  million  target  (see  Table  A-2):  they  did  less 
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well  on  the  A-10  airframe.  But  they  were  not  deemed  the  best  for  two 
reasons:  First,  the  predicted  availabilities  were  significantly  higher  than 
the  availabilities  actually  achieved  in  all  cases.  Second,  the  Bayesian 
procedure  is  significantly  more  complicated;  it  requires  estimation  of  a 
prior  distribution  by  combining  data  on  all  items  and  a  separate  Bayesian 
inference  for  each  item  in  each  period  with  historical  demand. 

Since  the  mean  tends  to  drift  with  time  and  we  are  unable  to  predict  the 
trend,  our  best  prediction  of  mean  demand  for  an  item  is  the  same, 
regardless  of  how  far  ahead  we  are  predicting.  This  implies  our  sixth 
finding. 

6.  The  estimate  of  variance  should  increase  as  the  period  being  predicted 
advances  into  the  future.  This  is  the  subject  of  Appendix  D,  where  VMRs 
are  developed  for  longer  prediction  periods.  From  a  theoretical  perspec¬ 
tive,  this  implies  that  a  stationary  compound  Poisson  process  —  used  in 
much  early  work  —  is  not  a  good  representation  of  the  physics.  Our  view  is 
that  we  have  a  Poisson  process,  but  the  mean  is  changing  over  time.  To 
compute  the  number  of  demands  during  a  given  period,  we  need  a  two- 
parameter,  discrete  probability  distribution,  with  a  variance  that  may 
exceed  the  mean.  For  this  purpose,  a  negative  binomial  is  a  convenient 
choice. 

7.  Different  values  of  the  exponential  smoothing  constant,  depending  on  the 
level  of  mean  demand,  do  not  improve  performance.  There  was  some 
suspicion  that  prediction  for  very  low-demand  items  could  be  improved  by 
incorporation  of  a  longer  history,  equivalent  to  a  smaller  constant  in  the 
exponential  smoothing.  In  our  research,  however  (described  in 
Appendix  E),  we  found  no  empirical  support  for  such  a  procedure. 
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CHAPTER  4 


CONCLUSIONS 


The  major  conclusions  are: 

1.  Demand  prediction  techniques  should  give  more  weight  to  recent  data, 
because  mean  demand  tends  to  drift  with  time. 

2.  Exponential  smoothing  is  consistently  the  best  technique  of  those  tested 
for  estimating  mean  demand.  With  quarterly  data,  a  smoothing  constant 
of  about  .4  appears  best. 

3.  Use  of  Poisson  or  a  constant  VMR  assumption  leads  to  poor  allocations  of 
investment,  because  system  availability  achieved  during  the  prediction 
period  is  low.  Across  the  system  studies,  a  "good”  choice  for  VMR  as  a 
function  of  the  annual  mean,  M,  is: 

VMR  =  1  +  .14M  3 

For  a  fixed  investment  of  $80  million  on  the  C-5  airframe,  the  Poisson 
assumption  leads  to  an  achieved  availability  of  .566;  the  formula  above 
leads  to  .828  for  the  next  year.  Alternatively,  the  same  availability  could 
be  achieved  with  20  percent  less  investment  and  the  VMR  formula  above. 
A  siightly  different  formula  for  VMR  is  recommended  if  the  prediction 
period  is  2  years  instead  of  one: 

VMR  =  1  +  .35M55 

Note  that  the  independent  variable,  M,  is  the  annual  mean,  rather  than 
the  mean  over  the  repair/resupply  time  as  used  by  AFLC.  It  is  the  drift  in 
mean  demand  over  the  prediction  period  that  is  important. 

4.  When  program  element  data,  such  as  flying  hours,  are  available  and 
demand  per  program  element  is  more  stable,  they  should  be  used  for 
prediction.  If  demand  per  quarter  is  more  stable  over  the  base  period,  that 
should  be  used,  instead.  In  our  analyses,  slightly  more  than  half  the  items 
were  in  the  former  group.  The  same  prediction  technique  is  recommended 
for  both  groups  of  items,  but  the  choice  of  technique  is  more  critical  for  the 
former  group. 

5.  Screening  techniques  should  be  used  to  eliminate  outliers.  This  includes 
cases  where  there  is  one  very  large  demand,  as  well  os  cases  in  which 
demands  have  been  very  high  and  a  zero  dx-mand  is  recorded  for  the  last 
period.  Specific  procedures  are  suggested  in  Chapter  3. 
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The  findings  below  are  less  definitive  than  the  conclusions,  because  they 
depend  on  the  techniques  used  in  this  study.  It  is  possible  that  other  researchers 
may  devise  new  techniques  that  will  alter  one  or  more  of  these  findings: 

1.  Techniques  for  predicting  trend  were  unsuccessful.  Since  mean  demand 
drifts  with  time,  this  implies  that  variance  increases  as  the  period  being 
predicted  moves  into  the  future  or  becomes  longer. 

2.  Combinations  of  techniques  did  not  improve  predictions.  An  example  of  a 
combination  technique  is  a  VMR  that  is  the  average  of  a  power  curve 
prediction  (Conclusion  3)  and  a  sample  estimate  based  on  historical  data 
for  the  item. 

3.  Bayesian  techniques  used  in  this  study  were  good  but  not  best.  The 
achieved  availabilities  were  fairly  high  in  most  cases,  but  the  predicted 
availabilities  were  almost  always  too  high. 

4.  A  different  exponential  smoothing  constant  for  low-demand  items  did  not 
improve  predictions.  There  had  been  some  suspicion  that  a  lower 
exponential  smoothing  constant,  equivalent  to  a  longer  history  period, 
might  improve  predictions  for  very  low-demand  items. 

These  conclusions  and  findings  appear  to  hold  consistently  across  the  C-5, 
A-10,  and  F-16  weapon  systems  as  represented  in  16  quarters  of  D041  data.  Further 
support  was  found  in  an  analysis  of  transaction  data  from  a  single  A-10  air 
base  -  England  Air  Force  Base  -  over  a  2-year  period.  Data  has  been  analyzed, 
both  with  and  without  program  element  information.  Predictions  have  been  made 
for  both  1-  and  2-year  periods.  We  believe  this  has  been  one  of  the  most 
comprehensive  demand  prediction  studies  undertaken  for  the  Air  Force  in  recent 
years. 

Some  words  of  caution  are  in  order.  There  are  a  few  items  with  very  large  and 
highly  variable  demand.  System  availability  is  critically  influenced  by  how  well  the 
prediction  technique  performs  on  those  items.  Unfortunately,  there  is  no  way 
around  this  fact  of  life,  which  seems  to  be  true  of  most  weapon  systems.  This  is  not  a 
defect  in  the  evaluation  measure;  rather,  it  is  related  to  the  physics  of  our  problem. 
The  practical  implications  are  that  the  technique  for  the  VMR  that  is  "best”  for  one 
weapon  system  is  not  likely  to  be  "best"  for  ail  others,  principally  because  of  the 
behavior  of  a  few  items.  However,  we  have  found  techniques  that  appear  to  do  a  good 
job  across  weapon  systems  and  to  be  substantially  better  than  techniques  now  in  use. 
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Another  point  of  caution  is  that  a  number  of  items  show  no  demand  over  the 
base  period.  Having  no  basis  for  picking  one  prediction  technique  over  another  for 
these  items,  we  have  excluded  them  from  predictions.  For  2-year  predictions,  the 
number  of  these  items  in  our  data  base  increases,  because  the  base  period  shrinks 
from  12  quarters  to  8. 

There  are  two  implications:  First,  it  is  desirable  to  retain  data  history  longer  to 
improve  detection  of  items  whose  "true”  mean  demand  is  not  zero.  Second,  it  may  be 
necessary  to  estimate  some  positive  demand  for  items  with  no  observed  demand  (e.g., 
using  a  Bayesian  procedure,  as  in  Feeney  and  Sherbrooke  [3].  If  this  is  not  done,  the 
resulting  value  for  weapon  system  availability  is  likely  to  be  unsatisfactory. 
Prediction  of  demand  for  items  that  show  no  past  demand  is  outside  the  scope  of  this 
report. 

We  have  attempted  to  understand  the  "physics”  of  the  demand  process  before 
comparing  different  techniques.  This  includes  the  statistical  work  that  describes  the 
drift  in  mean  demand  rate  over  time.  If  our  understanding  of  the  "physics”  is  good, 
the  results  should  have  wider  applicability.  However,  this  is  not  the  final  word  on 
demand  prediction. 
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APPENDIX  A 


DETAILS  OF  DEMAND  PREDICTION 


INTRODUCTION 

The  objective  in  this  appendix  is  to  provide  more  detailed  information  about 
the  demand  prediction  techniques  used  and  the  results  of  the  experiment.  For 
specified  budget  levels,  predicted  and  achieved  availabilities  are  shown  for  each 
technique  on  the  C-5  airframe  and  A-10  airframe.  The  technique  that  yielded  the 
best  results  was  also  the  best  in  some  preliminary  calculations  concerning  the 
F-16  engine/airframe. 

The  major  conclusion  of  relevance  to  this  report  is  that  a  relationship  of  the 

form: 


VMR  =  1  +  .14M’5  (Eq.  A  ll 

provides  the  "best”  estimate  of  variance-to-mean  ratio  (VMR)  for  an  item  over  the 
next  year,  based  on  an  estimate  of  its  annual  mean,  M.  Furthermore,  the  "best” 
estimate  of  annual  mean  demand  from  quarterly  data  was  obtained  with  exponenti 
smoothing  and  a  smoothing  constant  of  .4.  Additional  evidence  for  this  finding  is  to 
be  found  in  Sherbrooke  [2]. 

The  word  "best”  has  been  enclosed  in  quotes,  because  this  is  an  empirical 
finding  and  not  the  result  of  an  optimization.  It  is  possible  that  other  candidate 
techniques  could  improve  upon  the  procedures  we  recommend  for  estimating  means 
and  VMRs.  This  is  even  more  likely  with  different  data  sets,  particularly  if  data  is 
not  quarterly  or  if  predictions  are  desired  for  periods  substantially  longer  or  shorter 
than  a  year.  In  Appendix  D,  we  show  that,  for  demand  per  program  element  on  the 
F-16,  it  is  possible  to  find  a  somewhat  better  prediction  relationship  for  VMR  than 
Equation  A-l;  for  a  2-year  prediction  period,  too,  there  is  a  better  prediction 
relationship.  Nevertheless,  Equation  A-l  represents  a  substantial  improvement 
over  techniques  now  in  use. 


DEMAND  PREDICTION  TECHNIQUES 


\  fe  selected  17  candidate  techniques  that  had  shown  promise  in  earlier  studies. 
B>  a  "technique,”  we  mean  a  combination  of  a  procedure  for  estimating  the  annual 
mean  and  a  procedure  for  estimating  the  VMR.  For  example,  the  first  four 
techniques  in  Table  A-l  follow  the  same  "objective”  Bayesian  procedure  to  estimate 
the  VMR  but  different  procedures  to  estimate  the  mean  itself.  An  exponential 
smoothing  procedure  with  a  constant  of  .3  weights  the  most  recent  quarter  by  .3  and 
the  previous  estimate  of  the  mean  by  .7.  Repeated  application  yields  these  results:  a 
weight  of  .3  for  the  latest  quarter,  a  weight  of  .21  (.3  X  .7)  for  the  previous  quarter,  a 
weight  of  .147  (.21  X  .7)  for  the  previous  quarter,  and  so  on. 


TABLE  A-1 

DEMAND  PREDICTION  TECHNIQUES 


Technique 

VMR 

1  First  order  exponential  smoothing  using  3 

Bayes 

2  First  order  exponential  smoothing  using  4 

Bayes 

3  Last  quarter  (naive) 

Bayes 

4  8-quarter  moving  average 

Bayes 

5  First  order  exponential  smoothing  using  4 

1 

6  First  order  exponential  smoothing  using  4 

3 

7  First  order  exponential  smoothing  using  4 

1  +  14M58 

8.  8-quarter  moving  average 

1  +  14M58 

9  First  order  exponential  smoothing  using  4 

1  +  216M  69 

10  First  order  exponential  smoothing  using  4 

1  +  10M58 

1 1  First  order  exponential  smoothing  using  4 

1  +  14M-5 

1 2  First  order  exponential  smoothing  using  4 

1  +  8R 

1 3.  First  order  exponential  smoothing  using  4 

1  +  1  8R  58 

14  First  order  exponential  smoothing  using  4 

2  ♦  1  8R  58 

15  First  order  exponential  smoothing  using  4 

From  item  data 

16.  First  order  exponential  smoothing  using  4 

Average  of  Techniques  7  and  15 

17  Holt  linear  exponential  smoothing  with  4  and  5 

From  item  data 

A  2 


•->* «  ’j  -  » •  i* 


->  ■>  -  »  -  »  --V  JW  -N  --t  •>  --t  \ 


In  the  Bayesian  procedure,  the  mean  annual  demand  for  any  item  in  the  group 
of  items  on  a  weapon  system  is  assumed  to  have  the  same  gamma  prior  distribution. 
The  observed  demand  for  all  items  during  a  quarter  for  all  items  is  used  to  estimate 
the  two  parameters  of  the  gamma  prior  by  matching  the  first  two  moments  of  the 
distributions.  Then,  for  a  specific  item,  its  demand  is  combined  with  the  Bayesian 
prior  distribution  to  calculate  the  posterior  distribution  of  demand  for  that  item.  The 
mean  and  variance  of  that  item  posterior  distribution  are  used  as  the  two  parameters 
of  a  negative  binomial  demand  distribution  to  calculate  the  expected  backorders 
(EBOs)  for  the  item.  This  calculation  is  performed  for  each  quarter  independently, 
and  the  EBOs  are  combined  across  quarters  by  use  of  the  weights  from  exponential 
smoothing  or  the  moving  average.  This  "objective”  Bayesian  procedure  was  used 
successfully  in  the  original  tests  of  the  base  stockage  model,  the  forerunner  of 
METRIC,  and  is  described  in  Feeney  and  Sherbrooke  [3].  The  detailed  equations  are 
not  included  here,  because  the  Bayesian  procedure  is  not  recommended,  for  the 
reasons  described  below. 

One  departure  from  this  procedure  is  used  for  items  with  more  than 
1,200  demands  annually  during  the  first  12  quarters  of  demand  history.  For  the 
39  C-5  items  with  this  property,  the  Bayesian  procedure  underestimates 
significantly  the  true  mean  demand  (because  it  combines  the  information  on  all  560 
items).  For  these  items,  we  apply  Technique  1 1  —  the  same  technique  recommended 
in  Equation  A-l. 

Technique  5  is  the  constant,  Poisson  assumption;  Technique  6  is  a  constant 
VMR  of  3.  Techniques  7  through  11  are  variations  on  the  procedures  obtained  by 
nonlinear  estimation  applied  to  Air  Force  Logistics  Command  (AFLC)  data  in 
Sherbrooke  [2],  Techniques  12  through  14  are  based  on  the  repair/resupply  time 
estimate  of  mean  demand,  R.  Technique  15  estimates  the  VMR  for  the  item  from  the 
12  quarters  of  demand  history  on  the  item.  Technique  16  applies  the  results  of 
recent  forecasting  literature,  where  the  suggestion  is  made  that  forecasts  derived 
from  combinations  of  techniques  often  outperform  the  individual  techniques. 

Technique  17  is  the  only  one  that  attempts  to  predict  trend  in  the  mean 
demand.  It  combines  exponential  smoothing  with  a  second-trend  parameter  and  is 
described  in  Makridakis  and  Hibon  14],  To  prevent  extreme  predictions,  we  limited 
the  rate  of  increase  or  decrease  per  quarter  to  15  percent  of  the  mean  demand 
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predicted  from  simple  exponential  smoothing  with  a  constant  of  .4.  The  equations 
are  not  presented  here  because  the  results,  as  described  below,  were  extremely  poor. 

DEMAND  PREDICTION  RESULTS 

There  are  two  primary  considerations  in  evaluating  the  demand  prediction 
techniques.  The  most  important  consideration  is  to  obtain  a  high  achieved 
availability  during  the  prediction  year.  A  less  important  but  desirable  property  is  to 
have  a  predicted  availability  that  is  fairly  close  to  the  achieved  availability. 

Our  choice  is  Technique  11,  marked  with  asterisks  in  Tables  A-2  and  A-3, 
because  it  has  the  highest  achieved  availability  for  a  budget  of  $80  million  and  the 
fourth  highest  availability  for  a  budget  of  $100  million.  The  three  Bayesian 
techniques  that  finish  ahead  of  Technique  11  in  Table  A-3  are  overoptimistic;  their 
predicted  availabilities  always  exceed  the  achieved.  Technique  11  produces 
predicted  availabilities  that  are  reasonably  close  to  the  achieved  values.  The 
Bayesian  techniques  have  the  additional  disadvantages  of  being  significantly  more 
complex  and  unsuitable  in  an  initial  support  planning  context.  The  reason  is  that 
they  require  several  quarters  of  past  demand  data  to  estimate  VMR;  the  other 
techniques  require  only  an  estimate  of  the  mean. 

A  few  comments  should  be  made  about  the  other  techniques.  Technique  5,  the 
Poisson  assumption,  yields  especially  poor  results  particularly  as  we  move  to 
quarter  4;  Technique  6,  with  a  constant  VMR  of  3,  produces  better  though  similar 
results.  Not  surprisingly,  Techniques?  through  11,  as  a  group  produce  similar 
results.  Note  that  the  8-quarter  moving  average  -  Techniques  4  and  8  -  is  inferior  to 
exponential  smoothing  techniques  with  the  same  VMR  procedure.  It  is  a  little 
surprising  that  Technique  9  did  not  outperform  Technique  11,  because  nonlinear 
estimation  techniques  were  used  to  estimate  the  best  relationship  for  the  C-5  from 
its  own  data. 

Techniques  12  through  14  use  a  VMR  based  on  the  estimate  of  demand  over  the 
repair/resupply  time.  The  poor  performance  of  these  techniques  was  a  little  hard  to 
understand  at  first.  Since  the  mean  repair/resupply  times  vary  by  item  from  6  to 
57  days,  the  values  of  item  mean  repair  time  demand,  R,  are  much  smaller  than  the 


item  annual  demand,  M.  Accordingly,  larger  coefficients  are  needed  for  comparable 
variability.  However,  it  appears  that  these  techniques  are  inferior  to  Technique  11, 
because  it  is  not  the  variability  over  the  repair/resupply  time  that  is  important,  but 
the  drift  in  the  mean  demand  rate  over  the  year. 


TABLE  A-2 


ESTIMATED  AVAILABILITY  OF  C-5  AIRFRAMES:  S80  MILLION  BUDGET 


Availability 

Technique 

Cost 

($  million) 

Achieved 

Predicted 

Achieved  by  quarter 

annual 

annual 

1 

2 

3 

4 

1 

80  08 

.812 

856 

827 

856 

851 

.714 

2 

80.07 

.825 

846 

853 

.874 

858 

715 

3 

80  00 

.701 

.838 

853 

678 

698 

.576 

4 

80  06 

.686 

891 

687 

704 

723 

630 

5 

80.00 

566 

.986 

849 

.765 

631 

.020 

6 

80.18 

.678 

.758 

870 

.850 

833 

.160 

7 

8023 

810 

.641 

856 

841 

851 

690 

8 

80  00 

796 

.609 

826 

818 

871 

669 

9 

80  13 

746 

.419 

.742 

781 

816 

645 

10 

80  05 

822 

.740 

886 

842 

848 

.713 

*11* 

80  14 

*  828* 

771 

891 

852 

851 

.719 

12 

80  01 

752 

.731 

.743 

790 

828 

647 

13 

80.00 

.756 

724 

752 

.791 

828 

.652 

14 

8001 

710 

677 

698 

.744 

.780 

618 

15 

80  03 

776 

639 

789 

810 

848 

658 

16 

8002 

795 

606 

812 

834 

869 

667 

17 

— _ _ 

80.01 

693 

.707 

737 

.732 

758 

.543 

TABLE  A*3 


ESTIMATED  AVAILABILITY  OF  C-5  AIRFRAMl-S:  $100  MILLION  BUDGET 


Availability 

Technique 

Cost 

($  million) 

Achieved 

Predicted 

Achieved  by  quarter 

annual 

annual 

1 

2 

3 

4 

1 

100.19 

.925 

.946 

.981 

.974 

.957 

.787 

2 

100.05 

.917 

943 

.971 

.965 

.946 

.787 

3 

100.01 

.790 

.952 

.926 

.775 

.771 

687 

4 

100.07 

.924 

.958 

.945 

.976 

962 

.813 

5 

100.07 

622 

.999 

.919 

.858 

689 

.023 

6 

100.02 

.774 

.893 

.964 

.940 

924 

.267 

7 

100.00 

.905 

.820 

.952 

.930 

928 

.812 

8 

100.04 

.905 

.798 

.951 

.932 

.958 

.781 

9 

100  00 

.889 

612 

929 

.920 

.940 

.765 

10 

100.02 

.913 

892 

.964 

937 

.925 

.828 

*11* 

100.18 

*  913* 

.911 

.968 

.934 

919 

.831 

12 

100.00 

888 

844 

923 

.921 

.944 

765 

13 

100.03 

.896 

.838 

.934 

.930 

.949 

.771 

14 

100.02 

878 

.796 

.917 

.909 

.935 

750 

15 

100.07 

893 

812 

930 

.935 

.947 

.759 

16 

100.00 

.906 

790 

.948 

.942 

958 

774 

17 

100.04 

831 

880 

866 

865 

892 

703 

Technique  15  reinforces  the  results  of  other  studies  that  suggest  the 
impracticality  of  estimating  item  VMRs  from  past  history  on  the  item,  particularly 
since  the  item  means  are  drifting  over  time.  Technique  16  does  not  seem  to  be  an 
improvement  over  its  components.  Techniques  7  and  15.  Finally,  the  attempts  to 
predict  trend  in  the  mean  value  from  Technique  17  are  consistent  with  results 


reported  in  Sherbrooke  [2].  Though  demand  means  tend  to  drift  over  time,  we  have 
been  unabte  to  predict  the  direction  of  this  drift  for  individual  items. 

Similar  comparisons  were  made  on  the  480  items  of  the  A-10  airframe.  Ten  of 
the  more  promising  techniques  from  the  17  were  chosen  (see  Table  A-4). 


TABLE  A-4 

ESTIMATED  AVAILABILITY  OF  A-10  AIRFRAMES:  $80  MILLION  BUDGET 


Availability 

Technique 

Cost 

($  million) 

Achieved 

Predicted 

Achieved  by  quarter 

Achieved 

annual 

with 

cannibali- 

annual 

annual 

zation 

1 

2 

3 

1 

80  00 

892 

1.000 

.726 

.976 

973 

894 

2 

80  01 

927 

1.000 

.820 

.975 

983 

929 

3 

80.05 

.573 

1.000 

.406 

.952 

361 

.578 

4 

80  00 

.870 

1.000 

801 

.852 

.957 

906 

5 

80  07 

.966 

.979 

904 

.999 

.994 

966 

8B 

80.00 

703 

.988 

.500 

729 

880 

790 

*11* 

80.15 

*  977* 

972 

998 

963 

970 

980 

13 

80  02 

931 

617 

958 

945 

890 

961 

15 

80  01 

869 

735 

941 

845 

821 

930 

17 

80  08 

764 

604 

914 

.725 

653 

915 

Technique  8B  is  like  the  moving  average  Technique  8,  except  that  the  exponent 
of  .5  is  used,  making  the  VMR  expression  identical  with  Technique  11.  As  in  the  C  5 
case,  Technique  11,  which  is  exponential  smoothing  with  a  constant  of  .4  and  the 
power  curve  relationship  for  VMR,  is  the  best  technique,  as  indicated  by  the 
asterisks  in  the  table.  It  gives  the  highest  achieved  availability  of  .977  (only 
3  quarters  are  shown);  it  yields  the  highest  average  availability  of  .980  in  the  last 


column  when  full  cannibalization  —  that  is,  consolidation  of  shortages  on  the  fewest 
aircraft  possible  —  is  practiced. 

Some  of  the  major  results  in  Table  A-4  are  noted  here.  Technique  8B  (moving 
average)  is  far  inferior  to  exponential  smoothing  with  a  constant  of  .4.  (The  other 
moving  average,  Technique  4,  produces  the  worst  results  of  all  techniques.)  The 
agreement  between  predicted  and  achieved  availabilities  is  poor  for  all  techniques 
except  Technique  11,  and  the  second  best,  Technique  6,  which  is  a  VMR  of  3  on  all 
items. 

Table  A-5  provides  results  for  the  720  first-indenture  items  on  the 
F-16  engine/airframe.  The  same  10  techniques  were  compared,  and  Technique  11 
again  came  out  best.  It  has  a  slightly  lower  achieved  availability  than 
Technique  13,  but  the  predicted  availability  is  in  better  agreement.  We  note  that 
many  of  the  predicted  availabilities  are  too  high,  because  the  flying  hours  and 
demands  increased  substantially  over  the  16  quarters.  For  example,  the  demand 
rate  during  the  last  4  quarters  was  about  41  percent  greater  than  during  the  first  12 
quarters.  However,  excellent  availability  was  achieved  with  Technique  11,  when 
this  was  not  taken  into  account  (except  in  the  sense  that  exponential  smoothing 
weights  recent  data  more  heavily  in  estimating  the  mean  demand). 

Some  other  observations  concerning  Table  A-5  are  noted  here.  The  three 
Bayesian  techniques  (1,  2,  and  4)  and  the  Poisson  technique  (5)  produced  poor 
achieved  availabilities.  The  best  VMR  relationship  used  in  Technique  11  was  also 
used  in  Technique  8B.  The  former  employed  exponential  smoothing  with  a  constant 
of  .4;  the  latter  used  an  8-quarter  moving  average  with  far  inferior  results. 

In  the  F-16  case,  we  looked  at  the  items  that  were  primarily  responsible  for  the 
degradation  in  each  technique.  The  Bayesian  techniques  and  the  Poisson  did  badly 
on  five  or  six  items  that  showed  a  rapid  increase  in  usage  over  the  12  quarter  base 
period.  The  best  VMR  performed  much  better  on  these  items,  particularly  when  it 
was  used  with  the  more  responsive  exponential  smoothing  instead  of  an  8-quarter 
moving  average. 


The  F-16  differed  from  the  C-5  and  A- 10  in  that  the  demand  (and  the  flying- 
hour  program)  increased  substantially  over  the  16  quarters.  It  is  likely  that  demand 
per  flying  hour  will  be  more  stable  than  demand  itself.  A  more  detailed  analysis  of 
the  F-16  using  program  element  data  is  presented  in  Appendix  B. 

TABLE  A- 5 


ESTIMATED  AVAILABILITY  OF  F-16  ENGINES/AIRFRAMES:  $80  MILLION  BUDGET 


APPENDIX  B 


DEMAND  PREDICTION  USING  PROGRAM  ELEMENTS 

INTRODUCTION 

Here  we  compare  two  ways  to  predict  demand  and  weapon-system  availa¬ 
bility  —  the  present  Air  Force  Logistics  Command  (AFLC)  technique  and  the 
technique  recommended  in  this  report.  By  the  current  AFLC  demand  prediction 
technique,  we  use  of  an  8-quarter  moving  average  to  predict  the  mean  and  a 
variance-to-mean  ratio  (VMR)  given  by: 

VMR  =  1.1325R3<°7  [Eq.  B-l] 

where  R  is  the  predicted  mean  demand  over  the  repair/resupply  time  and  VMR  is 
constrained  to  be  no  less  than  1  and  no  more  than  5. 

The  technique  we  recommend  (Technique  11  in  Appendix  A)  is  exponential 
smoothing  with  a  smoothing  constant  of  .4  to  predict  the  mean  and  a  VMR  given  by: 

VMR  =  1  +  .14M-5  [Eq.B-2] 

where  M  is  the  predicted  annual  mean  demand  and  VMR  is  constrained  to  be  no  less 
than  1  and  no  more  than  20. 

RESULTS 

A  budget  of  $110  million  was  allocated  across  920  first-indenture  airframe  and 
engine  items  of  the  F-16,  using  an  optimal  availability  model  and  the  two  demand 
prediction  techniques  on  12  quarters  of  Recoverable  Consumption  Item  Require 
ments  (D041)  data.  The  technique  we  recommend  led  to  an  achieved  availability  of 
.433  over  the  4  quarters;  the  AFLC  procedure  led  to  only  .201.  This  comparison 
utilizes  program  element  information  on  items  whenever  the  information  is 
available;  this  is  particularly  important  on  such  systems  as  the  F-16,  where  demand 
changes  dramatically. 


With  a  budget  of  $180  million,  our  procedure  led  to  an  achieved  availability  of 
.704  versus  .408  for  the  AFLC  procedure.  Backorders  averaged  three  times  as  large 
under  the  AFLC  procedure. 

DESCRIPTION  OF  EXPERIMENT 

The  original  sample  of  1,214  items  from  the  D041  data  system  for  the  F-16 
engine/airframe  was  reduced  by  281  items  for  which  there  was  no  demand  during  the 
first  12  quarters.  Using  data  from  the  first  12  quarters  only,  we  divided  the 
remaining  933  items  into  two  groups: 

•  Group  A.  The  VMR  for  demand  was  smaller. 

•  Group  B.  The  VMR  for  demand  per  program  element  was  smaller. 

Of  the  933  items  in  Table  B-l,  demand  per  program  element  could  be  computed 
for  only  774,  because  the  program  element  (usually  flying  hours)  was  sometimes 
missing  from  the  item  data  record.  Of  these  items,  demand  per  program  element  had 
a  lower  VMR  over  the  first  12  quarters  on  511  items;  demand  on  the  remaining  were 
422  items. 


TABLE  B-1 

VMRs  FOR  ESTIMATORS  A  AND  B 


Estimator  A 

Estimator  B 

Best  A 

Best  B 

Number  of  items 

933 

774 

422 

511 

Average  VMR 

66  024 

43  683 

44  181 

34  323 

The  average  VMR  for  each  group  of  items  is  shown  at  the  bottom  of  Table  B-l.  Note 
that  these  VMRs  are  used  only  for  separating  the  items  between  Group  A  and 
Group  B,  not  for  predicting  demand.  The  reason  for  breaking  the  sample  into  two 
groups  was  to  use  demand  or  demand  per  program  element,  depending  on  stability; 
furthermore,  we  wanted  to  assess  the  prediction  accuracy  of  our  recommended 
procedure  and  the  AFLC  procedure  for  these  two  groups  separately.  The  procedure 


is  similar  to  that  used  in  Sherbrooke  [2],  where  the  computational  formulas  were 
derived. 

In  Table  B-2,  over  the  entire  group  of  933  items  with  some  positive  demand 
during  the  first  12  quarters,  average  demand  per  item  increased  from  22.0622  in 
quarter  1  to  50.9528  in  quarter  16  (131-percent  increase).  When  split  into  the  two 
groups  according  to  stability,  average  demand  per  item  for  Group  A  with  no  program 
element  increased  from  21.3483  in  quarter  1  to  30.3175  in  quarter  16  (42-percent 
increase);  the  average  demand  per  program  element  for  Group  B  increased  from 
27.0444/1,326  in  quarter  1  to  70.1919/2,154  in  quarter  16  (59-percent  increase).  Use 
of  the  program  element  does  tend  to  improve  stability  over  time. 

The  final  number  of  items  processed  was  reduced  to  a  multiple  of  20  for  ease  of 
processing,  i.e.,  420  items  for  Group  A  and  500  items  for  Group  B.  The  budget  of 
$110  million  was  divided  in  rough  proportion  to  the  number  in  each  group, 
$50  million  and  $60  million,  respectively.  (When  the  budget  was  $180  million,  the 
investment  was  split  equally.) 

Table  B-3  shows  the  results  for  items  in  Group  A  and  Table  B-4  the  results  for 
Group  B.  The  procedure  was  to  use  the  first  12  quarters  of  D041  data  to  predict  the 
mean  and  VMR  ratio  over  the  next  4  quarters,  as  described  above.  Then  an  optimal 
availability  model  was  used  to  allocate  investment  across  items.  Actual  demands 
during  the  next  4  quarters  were  to  assess  prediction  accuracy.  Briefly,  the  procedure 
was  to  determine  probabilistically  the  day  on  which  each  demand  occurred  and  the 
day  on  which  each  repair  was  completed.  The  availability  was  computed  across  the 
group  of  items  on  a  daily  basis  and  averaged  over  each  quarter. 

Achieved  availability  under  no  cannibalization  and  with  cannibalization  are 
shown  in  Tables  B-3  and  B-4.  The  average  for  the  year  is  shown,  as  well  as  the 
values  for  each  quarter.  The  average  number  of  backorders  is  more  than  twice  as 
large  in  each  table  where  the  AFLC  procedure  was  used.  In  Table  B-3,  AFLC 
availability  averages  .532.  compared  with  our  .746;  in  Table  B-4,  AFLC  availability 
averages  .378  versus  our  .581.  Multiplying  availabilities  to  obtain  the  system  values 
yields  .201  for  the  AFLC  procedure  and  .433  for  our  recommendation. 


TABLE  B-2 


AVERAGE  DEMAND/ITEM  BY  QUARTER  FOR  F-16 


Quarter 

Positive  demand 
in  12  quarters 

Estimator  #1  best 

Estimator  #2  best 

Average  demand/ 
item  by  quarter 

Average  demand/ 
item  by  quarter 

Flying  hour/item 
by  quarter 

1 

22.0622 

21.3483 

27  0444 

1,326 

2 

25.0054 

22.8839 

31.6505 

1,430 

3 

29  1629 

23.8294 

39.7060 

1,567 

4 

27  9378 

22.3389 

384273 

1,538 

5 

29  2765 

23.7749 

39  0995 

1,552 

6 

30.4373 

21  3152 

43  7991 

1,667 

7 

31  9625 

20.5190 

47  6622 

1,850 

8 

35.7728 

21.8175 

54.4347 

1,817 

9 

39.7717 

27  0142 

524633 

1,756 

10 

450772 

286422 

60  6680 

1,868 

11 

47  8617 

29  2062 

65  3131 

1,969 

12 

46.3810 

27.4360 

64  9737 

1,951 

13 

47  3655 

29.1043 

64.4646 

2,011 

14 

49  5531 

30  6682 

67.2545 

2,149 

15 

51  2776 

30  1872 

70  9152 

2,242 

16 

509528 

30  3175 

70  1919 

2,154 

Each  table  shows  some  detail  concerning  items  that  cause  backorders  and 
lower  availability  within  the  group.  We  have  printed  the  item  repair/resupply  time, 
the  prediction  of  mean  quarterly  demand  for  each  of  the  two  techniques,  the  VMR  for 
each  technique,  the  stock  level  for  each  technique,  th  actual  demands  in  the  last 
4  quarters,  and  the  average  backorders  existing  per  day  for  each  of  the  two 
techniques  on  the  item.  Only  those  items  for  which  the  sum  of  backorders  by  both 
techniques  exceeds  .5  are  printed;  the  other  items  do  not  degrade  availability. 
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TABLE  B-3 


GROUP  A:  DEMAND  PER  QUARTER/AVAILABILITY:  420  ITEMS 


Techniques 

11 

AFLC 

I ) 

AFIC 

NO 

No 

YPS 

Yes 

0  837 

0  637 

0909 

0  816 

0  678 

0  450 

0  843 

0  788 

0  807 

0  582 

0.904 

0  800 

0  622 

0  459 

0802 

0  795 

0  746 

0  S32 

0  86S 

0  800 

Predicted  mean 


AVERAGE  BACKORDERS  BV  ITEM  BY  PREDICTION  TECHNIQUES 


Predicted  vmr  Stock  level  Demand  m  quarters  i J  16 


11  AflC 


E»pected  backorders 


Sum 


TABLE  B-4 


GROUP  B:  DEMAND  PER  PROGRAM  ELEMENT/AVAILABILITY:  500  ITEMS 


Techniques 

11 

AFLC 

■  1 

AFIC 

Cannibalization 

NO 

NO 

Yes 

Yes 

Quarter 

1 

0  936 

0  841 

0  958 

0  899 

2 

0  878 

0  582 

0  949 

0  802 

3 

0  271 

0  066 

0  329 

0  165 

4 

0  239 

0.023 

0  3H 

0  133 

Average 

0  581 

0  3 ’8 

0  637 

3  500 

AVERAGE  BACKORDERS  BV  ITEM  BV  PREDICTION  TECHNIQUES 


Resupply 

time 

Predicted  mean 

Predicted  VMR 

5to<*  level 

Demand  m  quarters  13  - 

16 

E  «oe<  :eo  toac*  orders 

11 

AFLC 

11 

AFlC 

n 

AFLC 

13 

14 

15 

16 

n 

AFLC 

14 

4 

4 

2 

4 

3 

13 

12 

8 

30 

mm 

0  5 

8 

576 

367 

8 

H9 

74 

681 

827 

937 

813 

16  1 

»3 

9 

67 

70 

3 

1 1 

14 

92 

92 

72 

55 

3  1 

14 

9 

63 

67 

3 

10 

13 

70 

65 

74 

73 

WM 

0  i 

15 

9 

76 

76 

12 

15 

101 

106 

91 

112 

1  5 

0  5 

16 

9 

63 

58 

10 

'2 

86 

92 

90 

76 

09 

0  3 

30 

8 

119 

’28 

lilllili 

3 

21 

23 

178 

’85 

’96 

’56 

1  8 

1  0 

33 

3 

42 

21 

1 

8 

4 

219 

95 

24 

’  19 

0  5 

2  8 

48 

26 

2 

1 

1 

4 

3 

15 

9 

; 

4 

0  3 

0  7 

52 

5 

8 

6 

1 

4 

3 

1 

7 

'8 

•01 

06 

09 

59 

7 

5 

4 

2 

1 

4 

2 

5 

19 

’6 

50 

02 

06 

65 

20 

25 

’8 

2 

2 

1  7 

14 

38 

58 

3  3 

17 

0  2 

0  5 

n 

7 

2.898 

2.734 

ib 

5 

474 

346 

2  889 

2.267 

3.5  ’8 

3  b’8 

0  0 

•o  1 

74 

? 

1.789 

'  860 

'3 

5 

283 

240 

2.085 

2  '33 

2  120 

2  2  72 

3  0 

0  9 

78 

9 

246 

249 

5 

3 

57 

54 

228 

342 

30V 

432 

0  6 

:  9 

94 

19 

8 

5 

2 

1 

10 

7 

3 

5 

5 

40 

■ 

0  6 

104 

22 

22 

21 

2 

2 

13 

14 

28 

67 

55 

43 

1 1 

1  c 

112 

29 

’.757 

1.700 

'3 

S 

923 

763 

2.085 

2  1 30 

2  123 

2  295 

1 

7  ’ 

150 

19 

139 

81 

4 

3 

78 

51 

’83 

J 19 

331 

•54 

C  7 

•3  2 

162 

<8 

’06 

35 

4 

3 

57 

68 

77 

’’8 

282 

•84 

1  1 

0  ' 

174 

30 

15 

7 

2 

1 

21 

'2 

68 

24 

3 

28 

0  8 

2  5 

’82 

18 

b 

5 

2 

4 

4 

q 

15 

•2 

• 

0  3 

0  3 

2’2 

25 

’5 

9 

2 

2 

10 

8 

1  7 

25 

'  7 

25 

0  2 

J  6 

2’J 

’7 

260 

221 

6 

4 

H8 

'00 

3  76 

S42 

95  J 

872 

34  1 

44  ) 

2’7 

’6 

1  424 

’  4;a 

’2 

5 

44  J 

384 

’  35.* 

’  446 

2  094 

1  322 

0  0 

3  •’ 

2% 

26 

19 

21 

3 

2 

29 

. 

6  ! 

67 

’9 

3 

0  1 

107 

24 

0 

0 

t 

1 

2 

, 

8 

4 

0  1 

J  4 

13’ 

'8 
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; 

2 

i 
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11 

J1 

10 

49 
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148 

"I 

1 

4Q 

3 

2 

43 

»6 
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"3 

:a 

;  n 

’  8 

4C9 

1 

7 

2 

9 
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•9 

30 

;  3 

4  ’ 
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460 

1 

1 

24 

3 

2 

22 

1 ' 

57 

•’ 

V 

2  t 

4'0 

2  3b 

5 

3 

3) 

>8 

241 

;**6 

2  06 
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;  4 

47« 

1  7 

6 

2 

7 

5 

35 

8 

■ 
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:  s 
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Let  us  now  attempt  to  determine  what  accounts  for  the  degradation  in 
availability.  In  Table  B-3  item  #363  [national  stock  number  (NSN)  6605011685185] 
is  most  important  for  both  techniques.  The  item  costs  $89,000,  and  there  was  no 
program  element  in  the  data  record.  Our  technique,  as  compared  to  the  AFLC 
technique,  estimated  the  mean  to  be  twice  as  large  and  the  VMR  as  3  rather  than  2. 
Consequently,  our  stock  level  was  25  rather  than  15,  cutting  backorders  by  more 
that  a  factor  of  2.  There  were  several  other  items  in  Table  B-3  where  our  technique 
reduced  backorders  substantially,  mainly  because  of  the  more  reactive  estimate  of 
the  mean  (e.g.,  #50,  #120,  #224,  and  #284).  For  other  items,  a  combination  of  the 
mean  and  VMR  seems  to  be  responsible  (e.g.,  #49  and  #213).  Note  that  the  VMR  for 
our  recommended  procedure  exceeded  the  AFLC  cutoff  of  5  on  only  two  items  (#198 
and  #213).  Our  recommended  procedure  increases  the  backorders  on  some  items. 
However,  these  tend  to  be  items  with  fairly  few  backorders,  and  the  maximum 
increase  is  only  .6. 

Turning  to  Table  B-4,  we  find  that  item  #213  (NSN  2840011433254),  costing 
only  $568,  is  most  important  in  degrading  both  availabilities.  Our  technique 
predicts  a  larger  mean  and  VMR,  but  backorders  are  reduced  by  only  25  percent 
(most  of  the  backorders  occur  during  the  last  2  quarters).  The  AFLC  technique  has 
large  backorders  on  three  other  items  that  are  reduced  substantially  by  our 
technique  (e.g.,  item  #6  due  to  mean  and  VMR,  item  #73  due  primarily  to  the  VMR, 
and  item  #150  due  primarily  to  the  mean).  In  this  table,  our  technique  leads  to 
VMRs  as  high  as  16,  and  a  total  of  6  items  with  VMRs  in  excess  of  5.  Happily  our 
technique  does  not  increase  the  backorders  on  any  item  by  more  than  1,  a  desirable 
feature. 

One  technical  note  should  be  added.  The  VMRs  over  the  first  12  quarters 
(3  years)  exceed  44  for  demand  (Group  A)  and  34  for  demand  per  program  element 
(Group  B),  as  can  be  seen  from  the  bottom  ofTable  B-l.  By  contrast,  the  VMRs  in  our 
predictions  are  much  smaller  (exceeding  10  for  only  4  items).  The  reason  is  that  the 
VMR  in  the  predictions  is  only  over  a  year,  and  VMR  increases  with  the  length  of  the 
period. 

Similar  calculations  were  performed  for  the  A- 10  with  comparable  results.  For 
example,  on  the  320  items  where  demand  per  program  element  was  the  better 
predictor  the  average  availability  with  no  cannibalization  and  a  budget  of 
$50  million  was  .943  for  Technique  11  and  .855  for  the  AFLC  technique.  The 
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expected  backorders  were  5.9  and  19.6,  respectively.  These  values  should  be 
compared  with  Table  B-4  for  the  F-16. 

CORRELATIONS 

In  the  main  text,  we  have  calculated  the  correlations  between  demand  in 
2  periods  across  items.  It  was  noted  there  the  correlation  coefficient  is  not  affected  if 
demand  is  increasing  over  time  uniformly  across  all  items  (the  slope  of  the 
regression  line  is  affected,  but  not  the  correlation,  which  is  related  to  the  distance  of 
the  data  points  from  the  regression  line). 

However,  with  a  rapidly  increasing  program,  such  as  that  of  the  F  16,  it  is  of 
interest  to  compute  the  correlations  between  demand  per  program  element  since  the 
program  element  change  may  differ  from  item  to  item.  When  this  is  done  for  all 
615  items  on  the  F-16  with  program  element  information  in  each  of  the  16  quarters, 
the  results  are  shown  in  Table  B-5.  The  correlations  decrease  less  rapidly  than  those 
in  the  last  column  of  Table  3-1  in  Chapter  3,  but  the  results  are  similar. 

It  is  possible  to  compute  correlations  in  a  different  way  across  periods  for  an 
item.  The  difference  is  that  the  means  are  now  computed  from  an  individual  item, 
whereas  in  the  previous  correlations  the  means  were  computed  across  items  for  a 
specific  period.  With  16  observations,  there  are  15  intervals  of  length  1, 14  intervals 
of  length  2,  etc.  We  terminate  the  process  with  the  5  intervals  of  length  11,  beca  use 
the  sample  size  becomes  too  small  to  test  for  statistical  significance  after  that  point. 

Both  types  of  correlation  measure  the  degree  of  similarity  between 
observations  as  the  time  interval  between  observations  increases.  The  numerical 
values  will  be  very  different,  near  one  for  the  first  method  and  near  zero  for  the 
second.  The  advantage  of  the  first  method  is  that  the  results  for  longer  lags  can  be 
computed  and  compared. 


TABLE  B-5 


CORRELATIONS 

(Demand  per  program  element:  615  F-16  items) 


Number  of 
quarters 
apart 

Across  items 

Across  periods  by  item 

Correlation 

Number  of 
significant 
positive 
correlations 

Number  of 
significant 
negative 
correlations 

1 

.936 

.201 

151 

11 

2 

.903 

.085 

91 

18 

3 

.883 

.049 

225 

20 

4 

.888 

.029 

64 

26 

5 

883 

-.026 

39 

29 

6 

869 

-.030 

40 

33 

7 

.854 

-.034 

25 

28 

8 

.849 

.022 

53 

27 

9 

.845 

.013 

43 

24 

10 

829 

-.019 

33 

28 

11 

.807 

-.038 

25 

23 

12 

.810 

13 

826 

14 

825 

15 

749 

The  advantage  of  the  second  method  is  that  the  results  are  not  affected  by  the 
dispersion  of  mean  demand  levels  across  items.  Statistical  tests  of  significance  can 
therefore  be  employed.  Over  the  entire  group  of  615  items,  the  average  correlation  is 
significant  at  the  95  percent  level  for  intervals  of  lengths  1  through  3.  On  an 
individual  item  basis,  it  is  seen  that  many  items  have  a  significant  positive 
correlation  for  intervals  of  lengths  1  through  4.  After  that  point  (a  1-year 
separation),  the  number  of  positive  correlations  that  are  significant  tends  to  be  only 
slightly  larger  than  the  number  of  significant  negative  correlations. 

Table  B-6  shows  that  analysis  of  the  A- 10  aircraft  yields  similar  results.  Over 
the  entire  group  of  444  items,  the  average  correlation  is  significant  at  the  95  percent 
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level  for  intervals  of  lengths  1  through  4;  the  number  of  items  with  significant 
positive  correlation  is  also  substantially  greater  than  the  number  with  significant 
negative  correlation  for  intervals  1  through  4. 

TABLE  B-6 
CORRELATIONS 

(Demand  per  program  element:  444  A-10  items) 


Number  of 
quarters 
apart 

Across  items 

Across  periods  by  item 

Correlation 

Number  of 
significant 
positive 
correlations 

Number  of 
significant 
negative 
correlations 

1 

.921 

.207 

117 

3 

2 

.910 

095 

70 

13 

3 

897 

.064 

83 

5 

4 

886 

059 

48 

17 

5 

869 

001 

39 

22 

6 

872 

-.048 

19 

26 

7 

864 

-.022 

16 

17 

8 

849 

011 

22 

13 

9 

851 

011 

22 

11 

10 

844 

-  038 

18 

16 

11 

850 

-.056 

9 

18 

12 

847 

13 

857 

14 

862 

15 

822 

All  data  analyzed  in  this  report  have  been  quarterly  D041  data,  where  all  base 
demands  on  a  given  weapon  system  have  been  aggregated.  It  is  important  to  be  sure 
that  demand  at  an  individual  base  is  not  fundamentally  different  from  the  sum  of 
demand  at  all  bases;  that  demand  over  shorter  time  periods  does  not  behave 
differently.  For  this  reason,  we  obtained  the  transaction  data  from  a  base  for  a 
2  year  period.  The  base  was  England  Air  Force  Base,  Mississippi,  with  A- 10s.  and 
the  data  covered  the  period  from  April  1982  through  March  1984.  The  demands  for 
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each  item  (XD-2)  were  aggregated  in  2-week  periods  as  the  fundamental  unit  of 
analysis.  The  smallest  meaningful  period  would  have  been  a  week,  since  we  do  not 
want  the  lighter  flying  schedule  on  weekends  to  bias  the  results.  A  2-week  period 
was  selected  so  that  demands  during  the  period  would  be  slightly  larger;  this  gave  us 
52  periods  for  correlational  analysis. 

The  results  of  similar  correlational  analysis  on  the  2-week  demands  at  England 
Air  Force  Base  are  shown  in  Table  B-7.  Some  major  differences  are  noted  from  the 
previous  correlations.  The  correlations  decrease  fairly  uniformly  (only  the  first  20 
intervals  are  shown  in  the  table,  but  the  correlations  decrease  to  .201  for  the  largest 
separation  of  51  periods).  However,  the  first  value  of  .505  is  much  smaller  than  the 
first  value  in  the  other  correlational  analyses  across  items.  The  reason  is  that  the 
average  demands  per  period  are  much  smaller  and  the  variability  in  this  number 
across  items  is  much  smaller  (the  average  demand  per  item  2  weeks  at  England  Air 
Force  Base  is  about  .2  percent,  as  compared  with  about  35  percent  in  each  quarter 
across  all  F-16  bases). 

The  number  of  items  whose  correlations  are  positive  and  statistically 
significant  when  the  periods  are  adjoining  is  only  95/741,  or  12.8  percent,  as 
compared  with  about  25  percent  for  the  D041  data.  Again,  this  is  due  to  the  low 
values  of  demand  and  the  inability  to  obtain  statistical  significance.  However,  the 
qualitative  behavior  of  the  demand  for  2- week  periods  is  consistent  with  the  D041 
data,  providing  further  support  for  our  major  conclusions. 


TABLE  8-7 


CORRELATIONS  -  2-WEEK  PERIODS 
(Demand  per  program  element:  741  A-10  items) 


Across  periods  by  item 

Number  of 
quarters 
apart 

Across  items 

Correlation 

Number  of 
significant 
positive 
correlations 

Number  of 
significant 
negative 
correlations 

1 

.505 

.034 

95 

6 

2 

.478 

.027 

76 

1 

3 

AGC 

.“TVJ 

vt  1 

Ol 

i 

C 

-r 

4 

.443 

.013 

59 

5 

5 

.445 

.000 

43 

5 

6 

449 

.006 

45 

1 

7 

.427 

-  .003 

39 

3 

8 

.427 

-.004 

33 

0 

9 

.413 

-  007 

26 

2 

10 

.407 

-.015 

18 

3 

11 

395 

-.021 

22 

6 

12 

.387 

-.015 

27 

6 

13 

384 

-.007 

36 

7 

14 

.384 

-  017 

29 

2 

15 

389 

-  015 

22 

10 

16 

364 

-  024 

20 

9 

17 

373 

-  020 

29 

8 

18 

360 

-  024 

17 

8 

19 

.346 

-  .024 

12 

8 

20 

341 

-  020 

25 

10 
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APPENDIX  C 


PREDICTING  DIRECTION  OFTREND 


As  noted  in  the  main  text,  we  have  had  little  success  in  estimating  trend 
values.  A  simpler  problem  addressed  here  is  whether  we  can  differentiate  items  that 
are  likely  to  experience  heavier  demand  in  the  future  from  those  that  are  likely  to 
have  lighter  demand.  Predictions  of  trend  direction  based  on  demand  (without  the 
program  element)  for  the  A- 10  (shown  in  Table  C-l)  were  only  slightly  better  than 
random.  Initial  results  for  the  F-16  were  encouraging,  but  the  main  reason  was  the 
increasing  nature  of  the  F-16  flying-hour  program.  When  predictions  were 
attempted  on  the  basis  of  demand  per  program  element,  the  results,  as  displayed  in 
the  table,  were  very  poor. 

The  procedure  we  followed  was  to  estimate  the  mean  demand  in  the  future  by 
exponential  smoothing  with  a  constant  of  .4,  the  recommended  technique  from 
Appendix  A.  To  construct  a  sign  test,  we  then  compared  demands  in  the  12  base 
quarters  with  demands  in  the  quarters  that  immediately  precedes  them.  If  the 
second  quarter  was  larger  than  the  first,  +1  was  added  to  a  running  total.  If 
smaller,  - 1  was  added.  If  they  were  equal,  nothing  was  added.  Then,  following  the 
same  procedure,  we  compared  the  third  quarter  with  the  second,  and  the  process 
continued  until  the  eleventh  and  twelfth  quarters  were  compared.  The  result  was 
an  integer  statistic  with  values  that  could  range  from  —  1 1  to  +11. 

Our  hypothesis  was  that  the  true  mean  over  the  last  4  quarters  to  be  predicted 
was  likely  to  exceed  the  exponential  smoothing  prediction  when  this  statistic  was 
positive;  the  more  positive  the  statistic,  the  greater  our  confidence.  In  effect,  we 
were  hypothesizing  that  this  statistic  should  be  correlated  with  a  positive  trend, 
even  though  exponential  smoothing  gives  greater  weight  to  more  recent  data,  in  any 
case.  The  converse  proposition  is  made  with  respect  to  negative  values  of  the 
statistic. 

The  results  for  the  A-10,  using  demand,  and  for  the  F-16,  using  demand  per 
program  element,  are  listed  in  Table  C-l.  Values  of  the  statistic  less  than  —4  were 
combined  into  the  —4  category  (since  there  were  very  few),  and  values  greater  than 


+  4  were  combined  into  +  4.  Furthermore,  results  were  combined  for  absolute  values 
of  the  statistic,  since  we  are  likely  to  have  similar  confidence  with  a  value  of  +4  or 
—4,  though  our  inference  is  reversed.  Results  are  shown  cumulatively  for  other 
values  of  the  statistic;  e.g.,  3  includes  4,  on  the  assumption  that  if  we  draw  the 
inference  line  at  3,  we  want  to  include  3  and  any  larger  values. 


TABLE  C-1 

PREDICTION  OF  TREND  DIRECTION 


Criterion  value 

A- 10  demands  on  489  items 

4 

mm 

mm 

1-4 

Number  correct 

36 

86 

118 

209 

Number  of  items 

66 

160 

233 

407 

Percentage  correct 

54.5 

53.8 

50.6 

51.4 

F-16  demand  per  program  element  on  615  items 

Criterion  value 

4 

mm 

mm 

1  -4 

Number  correct 

22 

82 

108 

267 

Number  of  items 

47 

174 

229 

516 

Percentage  correct 

46  8 

47.1 

47  2 

51.7 

In  the  A- 10  case,  predictions  that  the  statistic  would  equal  or  exceed  3  in 
absolute  value  proved  accurate  slightly  more  than  50  percent  of  the  time,  but  these 
predictions  pertain  to  only  a  third  of  the  items. 

The  situation  is  the  reverse  for  the  F-16.  Here  the  predictions  for  negative 
values  when  the  criterion  was  applied  to  the  F-16  were  correct  more  than  61  percent 
of  the  time;  that  is,  demand  per  program  element  was  predicted  to  decline  for 
240  items,  and  147  of  these  predictions  proved  to  be  right.  This  was  balanced  by  the 
fact  that,  of  predictions  that  demand  per  program  element  would  rise  for  276  items, 
only  120  were  correct. 


We  tried  weighting  the  more  recent  values  of  the  statistic  more  heavily  and 
also  tried  using  the  quantitative  changes  between  quarters,  rather  than  + 1  or  —  1. 
The  results  were  similar  but  slightly  inferior. 


APPENDIX  D 


BEHAVIOR  OF  VARIANCE-TO-MEAN  RATIO  (VMR)  OVER  TIME 

VMR  PARAMETERS  THAT  CHANGE  BY  QUARTER 

In  an  earlier  report  on  demand  prediction  for  Air  Force  Logistics  Command 
(AFLC),  Sherbrooke  [2],  we  estimated  a  formula  for  the  VMR  as  a  function  of  the 
exponentially  smoothed  prediction  of  the  annual  mean,  M,  and  the  number  of 
quarters,  Q,  for  which  the  VMR  is  desired: 

VMR=1+AMB  [Eq.  Dll 

where  A  =  .141  +  .0125Q  and  B  =  .583  —  .0045Q.  The  estimation  was  based  on 
values  of  Q  from  4  to  8;  values  outside  that  range  are  extrapolations  that  should  be 
used  cautiously. 

An  alternative  formulation  that  may  be  more  useful  is  to  derive  expressions  for 
A  and  B  as  a  function  of  the  number  of  quarters  in  the  future.  For  example,  the  VMR 
over  the  next  year  for  a  specific  item  with  mean,  M,  is  not  constant,  as  might  be 
implied  by  Equation  D-l.  Our  research  shows  that  the  value  of  A  should  increase 
and  the  value  of  B  should  decrease  from  quarter  to  quarter. 

The  problem  is  that,  even  when  A  and  B  are  constants  rather  than  functions  of 
time,  the  optimal  values  can  vary  from  one  weapon  system  to  another.  Through 
most  of  this  report,  we  have  advocated  values  of  A  =  .14  and  B  =  .5.  These  values 
produce  much  better  availabilities  than  the  present  AFLC  techniques  and  a  variety 
of  alternatives  and  seem  to  perform  well  over  the  several  systems  tested.  But  the 
earlier  research  described  above  suggested  that  A  and  B  should  depend  on  the 
number  of  quarters  being  predicted:  in  particular,  for  demand  over  a  year  the  values 
would  be  A  =  .191  and  B  =  .565. 

In  the  specific  case  of  the  F-16,  the  latter  values  yield  availability  of  .780  versus 
.664  for  A  =  .14  and  B  =  .5  on  the  500  items  where  demand  per  program  element 
was  more  stable  and  availability  of  .865  versus  .840  on  the  420  items  where  demand 
was  more  stable.  To  test  the  way  in  which  VMR  should  change  with  time,  we  set  up 
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expressions  for  A  and  B  such  that  when  the  number  of  quarters  was  2.5  (halfway 
through  the  year),  the  values  would  agree  with  A  =  .191  and  B  =  .565.  The 
candidate  expression  tested  and  the  resulting  availabilities  for  the  500  items  with 
demand  per  program  element  are  shown  in  Table  D-l. 


TABLE  D-1 


AVAILABILITY  FOR  VARIOUS  VALUES  OF  A  AND  B 


A 

B 

Availability 

160  +  .0125  Q 

576  -  .0045  Q 

.785 

129  ♦  025 Q 

.5875  -  .009  Q 

.789 

129  +  025 Q 

.61  -  018  Q 

778 

066  +  .05  0 

6i  -  009 Q 

798 

066  ♦  .05  Q 

.5875  -  .009 Q 

.809 

Note  that  these  formulas  —  in  contrast  to  those  from  Sherbrooke  [2]  —  vary  by 
quarter  during  the  year;  during  the  first  quarter  a  value  of  Q  =  1  is  used;  during  the 
second,  a  value  of  Q  =  2;  during  the  third  quarter,  a  value  of  Q  =  3;  and  during  the 
fourth  quarter,  a  value  of  Q  =  4.  The  predicted  VMR  on  a  given  item  increases  from 
one  quarter  to  the  next. 

These  results  suggest  that  availability  can  be  improved  by  letting  the  values  of 
A  and  B  change  by  quarter.  The  impact  seems  to  be  fairly  modest,  though  we  have 
not  performed  an  exhaustive  search  to  determine  the  best  relationships  for  A  and  B. 
Such  a  search  has  not  been  conducted,  because  even  the  average  values  of  A  =  .191 
and  B  =  .565  have  not  been  established  as  "best"  for  the  F-16.  The  reason  for  the 
improvement  in  the  average  annual  availability  with  time-dependent  values  for  A 
and  B  is  primarily  the  higher  availabilities  that  prevail  during  the  third  and  fourth 
quarters. 

VMR  PARAMETERS  FOR  A  2-YEAR  PLANNING  HORIZON 

Throughout  this  report,  we  have  focused  on  a  planning  horizon  of  1  year  fur 
prediction.  Procurement  leadtimes  are  apt  to  be  longer  than  1  year,  and  the  question 
arises  whether  to  use  a  different  formula  for  VMR  when  prediction  over  longer 
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periods  is  required.  After  all,  we  showed  in  the  preceding  section  that  the  VMR 
should  increase  by  quarter  within  a  year,  though  the  effect  was  slight. 

Since  we  have  16  quarters  of  data,  it  is  possible  to  use  the  first  8  quarters 
(instead  of  12  quarters)  as  the  base  period  and  predict  over  the  last  8  quarters,  or 
2  years.  This  is  similar  to  the  procurement  leadtime  on  many  items.  Furthermore, 
the  current  Air  Force  procedure  uses  only  8  quarters  in  the  moving  average,  and 
exponential  smoothing  with  a  weight  of  .4  gives  negligible  weight  to  data  that  are 
more  than  8  quarters  old. 

The  analysis  reported  in  Appendix  B  was  repeated  for  a  2-year  prediction 
period.  The  AFLC  technique  of  Equation  B-l  was  again  compared  by  our 
recommended  technique  of  Equation  B-2  for  a  1-year  horizon.  Eight  other 
techniques  with  slightly  different  expressions  for  VMR  were  also  compared.  As 
before,  the  calculations  were  performed  for  the  F-16  and  A-10  with  separate 
computations  for  the  group  of  items  where  demand  per  program  element  was  more 
stable.  The  best  of  these  10  techniques  was: 

VMR  =  1  +  .35M55  ! Kq.  IJ-2] 

This  technique  performed  somewhat  better  than  other  techniques,  but  no  claim  of 
optimality  can  be  made. 

Only  summary  results  are  presented  here.  It  should  be  noted  that  the  number 
of  items  with  zero  demand  is  much  higher  than  in  Appendix  B,  because  there  is  only 
an  8-quarter  history  period.  For  the  same  reason,  the  number  of  items  for  which 
demand  per  quarter  is  more  stable  and  the  number  for  which  demand  per  program 
element  is  more  stable  differ  from  the  figures  in  Appendix  B. 

Note  that  the  budget  allocation  for  a  specific  weapon  system  between  the  group 
of  items  best  predicted  by  demand  per  quarter  and  the  group  by  demand  per  program 
element  is  not  optimal.  In  a  real  application,  all  items  for  a  weapon  system  should  be 
run  at  one  time  so  that  the  same  Lagrange  multiplier  is  used.  We  have  separated 
them  here,  for  analytical  purposes,  to  study  the  behavior  of  each  technique  on  each 
group.  An  important  conclusion  that  emerges  here,  as  well  as  in  Appendix  B,  is  that 
the  same  technique  should  be  used  on  both  groups  of  items. 


The  middle  part  of  Table  D-2  shows  the  average  availabilities  achieved  over 
the  2-year  prediction  period  under  the  condition  of  no  cannibalization;  the  lower  part 
shows  the  availabilities  with  cannibalization,  or  consolidation  of  part  shortages  on 
the  fewest  end  items. 

Clearly,  our  recommended  prediction  —  Technique  11  from  Appendix  B  —  is 
generally  much  better  than  the  AFLC  technique,  and  our  new  technique  from 
Equation  D-2  is  better  still  for  the  2-year  prediction.  This  result  is  true,  with  or 
without  cannibalization.  The  largest  improvements  are  found  in  the  group  of  items 
that  are  best  predicted  by  demand  per  program  element. 


TABLE  D-2 


Number  of  items 
Budget  ($  million) 


Eq.  B-1  -  AFLC 
Eq  B-2  -  11 
Eq  D-1  -  New 


2-YEAR  PREDICTIONS 


F-16 


Demand  per 
quarter 


Demand  per 
program 
element 


Demand  per 
quarter 


Demand  per 
program 
element 


Achieved  availability  with  no  cannibalization 


Demand  per 
quarter 


Demand  per 
program 
element 


Demand  per 
quarter 


Demand  per 
program 
element 


Achieved  availability  with  cannibalization 


APPENDIX  E 


LOWER  SMOOTHING  CONSTANT  FOR  LOW-DEMAND  ITEMS 


There  was  some  suspicion  that  predictions  for  very  low-demand  items  could  be 
improved  by  incorporation  of  a  longer  history,  equivalent  to  a  smaller  constant  in  the 
exponential  smoothing.  Our  procedure  was  to  use  a  smoothing  constant  of  .1  on  any 
items  for  which  demands  numbered  fewer  than  10  demands  in  the  first  12  quarters 
(typically  about  20  percent  of  the  items).  Though  the  predicted  availabilities  were 
very  slightly  higher,  the  actual  achieved  availabilities  were  slightly  higher  in  only 
one  case  and  lower  in  seven.  This  is  shown  in  Table  E-l.  It  should  be  noted  that  the 
values  of  .1  and  10  demands  are  arbitrary.  Different  values  may  give  somewhat 
different  results. 

In  Table  E-l,  the  achieved  availabilities  are  generally  higher  than  the 
predicted  availabilities  for  the  C-5  engine,  lower  for  the  A-10  and  F-16.  The  reason 
is  that  the  annual  dollar  value  of  demand  for  the  C-5  engine  decreased  during  the 
period  being  predicted  to  only  62.6  percent  of  its  value  in  the  base  period,  while  the 
value  increased  19.5  percent  for  the  A-10  and  increased  41.4  percent  for  the  F-16. 
However,  the  allocation  of  investment  appears  to  have  been  very  good.  Results  for 
an  8-quarter  moving  average  are  shown  at  the  highest  budget  level  for  each  weapon 
system.  Note  that  in  two  of  the  three  cases,  the  moving  average  leads  to  severe 
degradation  in  achieved  availability. 

In  summary,  Technique  11  with  a  smoothing  constant  of  .4  for  all  items 
appears  to  be  best. 


K  l 


TABLE  E-1 


AVAILABILITIES  WHEN  DIFFERENT  SMOOTHING  CONSTANTS  ARE  USED 


C-5  Engine 

Smoothing  constant 

4  for  all  items 

1  for  items  with  fewer 

than  10  demands 

8udget  (S  million) 

Predicted 

Achieved 

Predicted 

Achieved 

10 

614 

.796 

■ 

796 

15 

803 

878 

878 

20 

917 

9SS 

917 

956 

25 

972 

.977 

972 

974 

25 

97SJ 

978a 

A-10  Airframe 

Smoothing  constant 

4  for  all  items 

1  for  items  with  fewer 

than  lOdemands 

Budget  (S  million) 

Predicted 

Achieved 

Predicted 

Achieved 

40 

205 

309 

207 

303 

60 

788 

847 

789 

844 

80 

972 

96  2 

972 

960 

100 

997 

987 

997 

987 

100 

999, 

601- 

F-16  Engine/Airframe 

Smoothing  constant 

4  for  all  items 

1  for  items  with  fewer 

than  10  demands 

Budget  (5  million) 

Predicted 

Achieved 

Predicted 

Ach.eved 

60 

359 

225 

361 

224 

90 

808 

733 

809 

729 

120 

962 

927 

963 

925 

150 

994 

974 

994 

974 

150 

998’ 

392* 

— 

4  Indicates  tr.«  result  of  an  8  quarter  moving  average  to  predict  the  mean  rathe'  than  exponential 
smoothing 
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